Tensor Decomposition for Model Reduction in Neural Networks: A Review [Feature]

Modern neural networks have revolutionized the fields of computer vision (CV) and Natural Language Processing (NLP). They are widely used for solving complex CV tasks NLP such as image classification, generation, machine translation. Most state-of-the-art over-parameterized require a high computational cost. One straightforward solution is to replace layers with their low-rank tensor approximations using different decomposition methods. This paper reviews six methods illustrates ability compress model parameters convolutional (CNNs), recurrent (RNNs) Transformers. The accuracy some compressed models can be higher than original versions. Evaluations indicate that decompositions achieve significant reductions in size, run-time energy consumption, well suited implementing on edge devices.